Massively parallel expressed sequence tag clustering

Expressed Sequence Tag (EST) sequencing is a highly efficient technique that samples expressed genes required for most cellular functions. While this is a well-studied problem and many software tools have been developed, large-scale EST clustering has previously been pursued through incremental approaches, a pipeline of programs and manual efforts to achieve a modest degree of parallelism. Here, we present the first method that can directly cluster millions of ESTs on thousands of processors. This approach requires only linear space and uses rigorous alignment-based techniques to ensure biological accuracy. Further, we minimize computationally intensive alignments with single linkage clustering and develop a method to limit the formation of large spurious clusters. The computational scalability and biological validity of this approach is demonstrated by clustering mouse EST data, one of the two largest EST collections, on a 1,024 node BlueGene/L supercomputer.